Interoperability, as understood herein, refers to the ability of a system, product, or application to work with other systems, products, or applications without special effort on the part of the customer. Although many applications exist which can expose an external graphical user interface (GUI) for interoperability via an application programming interface (API) or built on the basis of one or more software development kits (SDKs) which allow interaction with the GUI by various low-level programming techniques, there exist a wide range of applications which cannot be connected by any existing technique.
An example of an application which often cannot be connected by standard API and/or SDK techniques as described above is an application running in a remote environment (e.g., in a server/client architecture). In this case, the user (client) may be able to see an image of an application on the screen of the client device and perform various input operations using a keyboard and/or mouse, but existing application integration techniques cannot recognize user interface (UI) elements or connect to any API exposed by the application, even when such APIs exist and are ready to use. Examples of remote environments in which this problem is prevalent include Microsoft Remote Desktop, Citrix XenApp, PCAnywhere, Oracle VM, etc.
Commonly, this kind of interoperability problem can be described by or attributed to one or more of the following issues: lack of a reliable connector for an interaction with such applications; there is no object model exposed by GUI elements or objects (for example, buttons, list boxes, links, tables, etc.) of these application types; and lack of a satisfactory algorithm for recognizing GUI elements at run-time.
One present attempt at solving the interoperability issue includes technology focused on Optical Mark Recognition (OMR), which is used with Optical Character Recognition (OCR) engines to format text while generating a specific text document. OMR also provides the ability to recognize a text document with a template designed by Developers. However, OMR and similar technologies do not provide an ability to interact with an application image, among other deficiencies. Presently, a great deal of research and development is needed, and tremendous effort is required to be invested, to find methods of connectivity, if possible, for even simple interactions with many target applications (e.g., OCR, mouse click, etc.).